Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GCP support for express cli #187

Merged
merged 32 commits into from
Aug 2, 2023

Conversation

gangadhar3303
Copy link
Contributor

@gangadhar3303 gangadhar3303 commented Jun 15, 2023

Description

This pull request adds support for GCP in express-cli. Users can now provision the infra in either AWS or GCP using the --init option with the values aws or gcp.

To identify which cloud the user has provisioned, we leverage the file remote-setup-config.yaml. When the user runs --start, we append the cloud details to remote-setup-config.yaml as shown below:

For GCP

devnetRegion: us-east1
cloud: gcp

For AWS:

devnetRegion: eu-west-1
cloud: aws

Changes

  • Add multi-cloud support for --init option.
  • Add GCP and AWS tf variables to env.example
  • Updated remote-setup-config.yaml to include cloud details
  • Updated start/stop instances scripts to support both AWS and GCP.
  • Restructed terraform files.
    terraform
    ├── aws           # Terraform code for AWS resources
    ├── gcp           # Terraform code for GCP resources
    └── variables
        ├── aws_vars.tf     # Variables specific for AWS
        ├── common_vars.tf  # Variables common for AWS & GCP
        └── gcp_vars.tf     # Variables specific for GCP
    
  • Renamed option --aws-key-add to --ssh-key-add and updated relative js files to handle both AWS and GCP

Breaking changes

During our testing, we did not observe any breaking changes resulting from the current modifications. However, it appears that few functionalities breaking due to existing code. (note: these issues are not consistently reproducible.)

  • SSH is failing after restarting sshd
  • Stopping and starting heimdall service is hanging

Checklist

  • I have added at least 2 reviewer or the whole pos-v1 team
  • I have added sufficient documentation in code
  • I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply

Cross repository changes

None

Testing

  • I have tested this code manually on local environment
  • I have tested this code manually on remote devnet using express-cli
  • I have tested this code manually on mumbai

@gangadhar3303 gangadhar3303 marked this pull request as draft June 15, 2023 04:30
Copy link
Contributor

@marcello33 marcello33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR 🙌
Despite being a draft, I left some preliminary comments to help you solve some issues I see before getting to a finalised version, and also to start a conversation on some open points.

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
src/express/commands/aws-instances-start.js Outdated Show resolved Hide resolved
src/express/commands/aws-keypair-add.js Outdated Show resolved Hide resolved
src/express/commands/setup-datadog.js Outdated Show resolved Hide resolved
src/express/common/config-utils.js Outdated Show resolved Hide resolved
src/express/common/config-utils.js Show resolved Hide resolved
terraform/gcp/.env.example Outdated Show resolved Hide resolved
terraform/gcp/main.tf Show resolved Hide resolved
terraform/gcp/variables.tf Outdated Show resolved Hide resolved
@gangadhar3303 gangadhar3303 marked this pull request as ready for review July 4, 2023 10:23
@gangadhar3303 gangadhar3303 changed the title [WIP/Draft] Add GCP support for express cli Add GCP support for express cli Jul 4, 2023
src/express/commands/keypair-add.js Outdated Show resolved Hide resolved
src/express/commands/setup-datadog.js Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
src/express-cli.js Outdated Show resolved Hide resolved
src/express/commands/aws-instances-start.js Outdated Show resolved Hide resolved
src/express/common/config-utils.js Outdated Show resolved Hide resolved
src/express/common/config-utils.js Outdated Show resolved Hide resolved
src/express/common/constants.js Outdated Show resolved Hide resolved
src/express/common/constants.js Outdated Show resolved Hide resolved
src/express/common/constants.js Outdated Show resolved Hide resolved
Copy link
Contributor

@marcello33 marcello33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve the (minor) comments.
Else, LGTM, thanks

@marcello33
Copy link
Contributor

Also, please solve prettier and linter issues, thanks

@marcello33
Copy link
Contributor

There's a typo in the PR description
It should be

For GCP

devnetRegion: us-east1
cloud: gcp

For AWS:

devnetRegion: eu-west-1
cloud: aws

@gangadhar3303
Copy link
Contributor Author

There's a typo in the PR description It should be

For GCP

devnetRegion: us-east1
cloud: gcp

For AWS:

devnetRegion: eu-west-1
cloud: aws

Updated

@marcello33
Copy link
Contributor

There's a typo in the PR description It should be

For GCP

devnetRegion: us-east1
cloud: gcp

For AWS:

devnetRegion: eu-west-1
cloud: aws

Updated

Please also update region > eu-west-1

@marcello33
Copy link
Contributor

Code LGTM, hence approved. There might be some more changes required after testing.
Thanks

.env.example Outdated Show resolved Hide resolved
.env.example Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
src/express/common/config-utils.js Outdated Show resolved Hide resolved
terraform/variables/aws_vars.tf Outdated Show resolved Hide resolved
terraform/variables/aws_vars.tf Outdated Show resolved Hide resolved
terraform/variables/gcp_vars.tf Outdated Show resolved Hide resolved
terraform/variables/gcp_vars.tf Outdated Show resolved Hide resolved
.env.example Outdated Show resolved Hide resolved
@anshalshukla
Copy link
Contributor

anshalshukla commented Jul 24, 2023

The setup fails in case of GCP with the following error:

❌ Command
 `cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml`
 failed too many times with error :
 Error: Command failed: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit
    at makeError (/Users/ashukla/polygon/matic-cli/node_modules/execa/index.js:174:9)
    at /Users/ashukla/polygon/matic-cli/node_modules/execa/index.js:278:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async runSshCommand (/Users/ashukla/polygon/matic-cli/src/express/common/remote-worker.js:21:5)
    at async runSshCommand (/Users/ashukla/polygon/matic-cli/src/express/common/remote-worker.js:41:7)
    at async runSshCommand (/Users/ashukla/polygon/matic-cli/src/express/common/remote-worker.js:41:7)
    at async runRemoteSetupWithMaticCLI (/Users/ashukla/polygon/matic-cli/src/express/commands/start.js:448:3)
    at async start (/Users/ashukla/polygon/matic-cli/src/express/commands/start.js:513:5)
    at async Proxy.cli (/Users/ashukla/polygon/matic-cli/src/express-cli.js:195:5) {
  code: 1,
  stdout: null,
  stderr: null,
  failed: true,
  signal: null,
  cmd: 'ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit',
  timedOut: false,
  killed: false
}

The command ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit works partially (ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected]) if I try to run this command manually on cli, the subsequent commands I believe fails because of missing quotes and tries to perform the cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit locally (it should be performing this on the remote instance after ssh).

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] 'cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit' executes the commands subsequent to ssh on the remote machine. While this is still the case when we try to run the setup on aws but it is failing on gcp.

@anshalshukla
Copy link
Contributor

anshalshukla commented Jul 24, 2023

We need to update the documentation as well. There aren't any instructions on how we can install and configure gcloud (init, auth application-default login).

.env.example Outdated Show resolved Hide resolved
@gangadhar3303
Copy link
Contributor Author

The setup fails in case of GCP with the following error:

❌ Command
 `cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml`
 failed too many times with error :
 Error: Command failed: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit
    at makeError (/Users/ashukla/polygon/matic-cli/node_modules/execa/index.js:174:9)
    at /Users/ashukla/polygon/matic-cli/node_modules/execa/index.js:278:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async runSshCommand (/Users/ashukla/polygon/matic-cli/src/express/common/remote-worker.js:21:5)
    at async runSshCommand (/Users/ashukla/polygon/matic-cli/src/express/common/remote-worker.js:41:7)
    at async runSshCommand (/Users/ashukla/polygon/matic-cli/src/express/common/remote-worker.js:41:7)
    at async runRemoteSetupWithMaticCLI (/Users/ashukla/polygon/matic-cli/src/express/commands/start.js:448:3)
    at async start (/Users/ashukla/polygon/matic-cli/src/express/commands/start.js:513:5)
    at async Proxy.cli (/Users/ashukla/polygon/matic-cli/src/express-cli.js:195:5) {
  code: 1,
  stdout: null,
  stderr: null,
  failed: true,
  signal: null,
  cmd: 'ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit',
  timedOut: false,
  killed: false
}

The command ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit works partially (ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected]) if I try to run this command manually on cli, the subsequent commands I believe fails because of missing quotes and tries to perform the cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit locally (it should be performing this on the remote instance after ssh).

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /Users/ashukla/.ssh/id_rsa.pem [email protected] 'cd ~/matic-cli/devnet && ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml && exit' executes the commands subsequent to ssh on the remote machine. While this is still the case when we try to run the setup on aws but it is failing on gcp.

It's running fine for us
image

@anshalshukla
Copy link
Contributor

anshalshukla commented Jul 26, 2023

Can you share your .env file. Also for setting up gcloud cli I've used

gcloud init
gcloud auth application-default login

Is this correct or is there any other way to setup gcloud on the local machine?

For your reference this is my .env file

# Terraform variables - AWS specific
TF_VAR_AWS_PROFILE=default # do not change this
TF_VAR_AWS_REGION=eu-west-1 # AWS region to deploy to (default is eu-west-1)
TF_VAR_INSTANCE_AMI=ami-01dd271720c1ba44f # ami instance type (default is ami-01dd271720c1ba44f, running ubuntu 22.04)
TF_VAR_BOR_IOPS=3000 # Amount of provisioned IOPS for Bor client
TF_VAR_ERIGON_IOPS=3000 # Amount of provisioned IOPS for Erigon client
TF_VAR_BOR_ARCHIVE_IOPS=3000 # Amount of provisioned IOPS in Bor archive node
TF_VAR_ERIGON_ARCHIVE_IOPS=3000 # Amount of provisioned IOPS in Erigon archive node
TF_VAR_PEM_FILE=aws-key # name of the certificate (.pem file) previously generated (default is aws-key)
TF_VAR_BOR_INSTANCE_TYPE=t2.xlarge # type of the EC2 VM instance for Bor client (default is t2.xlarge)
TF_VAR_ERIGON_INSTANCE_TYPE=r5b.large # type of the EC2 VM instance for Erigon client (default is r5b.large)
TF_VAR_BOR_ARCHIVE_INSTANCE_TYPE=t2.xlarge # default t2.xlarge
TF_VAR_ERIGON_ARCHIVE_INSTANCE_TYPE=r5b.large # default r5b.large
TF_VAR_BOR_VOLUME_TYPE=gp3 # default gp3
TF_VAR_ERIGON_VOLUME_TYPE=gp3 # default gp3
TF_VAR_BOR_ARCHIVE_VOLUME_TYPE=io1 # type of EBS volume for Bor archive nodes (default is io1)
TF_VAR_ERIGON_ARCHIVE_VOLUME_TYPE=io1 # type of EBS volume for Erigon archive nodes (default is io1)

# Terraform variables - GCP specific
TF_VAR_PROJECT_ID=prj-polygonlabs-pos-v1-dev # GCP Project ID
TF_VAR_GCP_REGION=europe-west2  # GCP region to deploy to (default is europe-west2)
TF_VAR_ZONE=europe-west2-a # Zone to provision the GCP compute resources
TF_VAR_SUBNET_CIDR_RANGE=10.2.0.0/16
TF_VAR_FW_RULE_SUFFIX=matic # Suffix to be used in firewall rule name.
TF_VAR_INSTANCE_IMAGE=ubuntu-2204-jammy-v20230302 #  instance image os (default is ubuntu-2204-jammy-v20230302, running ubuntu 22.04)
TF_VAR_GCE_PUB_KEY_FILE=/Users/ashukla/.ssh/id_rsa.pub # absolute path pointing to the public key file
TF_VAR_USER=ubuntu 
TF_VAR_BOR_MACHINE_TYPE=n2d-standard-4  # type of the EC2 VM instance for Bor client (default is n2d-standard-4)
TF_VAR_ERIGON_MACHINE_TYPE=n2d-standard-4  # type of the EC2 VM instance for Erigon client (default is n2d-standard-4)
TF_VAR_BOR_ARCHIVE_MACHINE_TYPE=n2d-standard-4  # default n2d-standard-4 
TF_VAR_ERIGON_ARCHIVE_MACHINE_TYPE=n2d-standard-4  # default n2d-standard-4
TF_VAR_BOR_VOLUME_TYPE_GCP=pd-ssd # default pd-ssd
TF_VAR_ERIGON_VOLUME_TYPE_GCP=pd-ssd # default pd-ssd
TF_VAR_BOR_ARCHIVE_VOLUME_TYPE_GCP=pd-balanced # type of EBS volume for Bor archive nodes (default is pd-balanced)
TF_VAR_ERIGON_ARCHIVE_VOLUME_TYPE_GCP=pd-balanced # type of EBS volume for Erigon archive nodes (default is pd-balanced)

# Terraform variables - Common for both AWS and GCP
TF_VAR_VM_NAME=hellogcp-ubuntu # default "polygon-user". It can be any string, used to discriminate between instances
TF_VAR_DOCKERIZED=no # default "no", otherwise only one VM is created and the Polygon devnet will run in docker containers
TF_VAR_BOR_DISK_SIZE_GB=20 # size of the disk in GB for Bor client (default is 20GB)
TF_VAR_ERIGON_DISK_SIZE_GB=20 # size of the disk in GB for Erigon client (default is 20GB)
TF_VAR_BOR_ARCHIVE_DISK_SIZE_GB=100 # size of the disk in GB in Bor archive node (default is 100GB)
TF_VAR_ERIGON_ARCHIVE_DISK_SIZE_GB=100 # size of the disk in GB in Erigon archive node (default is 100GB)
TF_VAR_BOR_VALIDATOR_COUNT=2 # number of Bor validator nodes (default is 2). Note that while spinning up a public network (mainnet/mumbai) node, this will serve as a non-validator
TF_VAR_BOR_SENTRY_COUNT=1 # number of non-validator Bor sentry nodes (default is 1)
TF_VAR_BOR_ARCHIVE_COUNT=1 # number of Bor archive nodes (default is 0)
TF_VAR_ERIGON_VALIDATOR_COUNT=0 # number of Erigon validator nodes (default is 0). Note that while spinning up a public network (mainnet/mumbai) node, this will serve as a non-validator
TF_VAR_ERIGON_SENTRY_COUNT=0 # number of non-validator Erigon sentry nodes (default is 0)
TF_VAR_ERIGON_ARCHIVE_COUNT=0 # number of Erigon archive nodes (default is 0)
TF_VAR_INSTANCE_AMI=ami-01dd271720c1ba44f # ami instance type (default is ami-017fecd1353bcc96e, running ubuntu 22.04)
TF_VAR_PEM_FILE=id_rsa # name of the certificate (.pem file) previously generated (default is aws-key)
PEM_FILE_PATH=/Users/ashukla/.ssh/id_rsa.pem # absolute path pointing to the certificate previously downloaded

# Polygon network based variables (see configs/README.md) for more detailed info
DEFAULT_STAKE=10000 # default stake for each validator (in matic)
DEFAULT_FEE=2000 # default amount of fee to topup heimdall validator
NETWORK= # mention "mumbai", "mainnet" or leave empty if spinning up a local devnet
HEIMDALL_SEEDS=e72c0466a02ea43b2198bd3a9454b87a3ef0d77e@54.147.31.250:26656,[email protected]:26656 # Heimdall seeds to be added in config.toml (when running a mainnet/mumbai node )
BOR_CHAIN_ID= # bor chainID (leave empty to get a random one)
HEIMDALL_CHAIN_ID= # heimdall chainID (leave empty to get a random one)
SPRINT_SIZE=64 # sprint size (number of blocks for each bor sprint)
SPRINT_SIZE_BLOCK_NUMBER=0 # comma separated values defining the block heights of bor where sprint length must change
BLOCK_NUMBER=0 # comma separated values defining the block heights of bor where block time must change
BLOCK_TIME=2 # comma separated values defining the block times for the relative BLOCK_NUMBERs
BOR_REPO="https://github.com/maticnetwork/bor.git" # repo of bor project
BOR_BRANCH=develop # bor branch defining the version to be used in the devnet
ERIGON_REPO="https://github.com/ledgerwatch/erigon.git" # repo of Erigon project
ERIGON_BRANCH=devel # Erigon branch defining the version to be used in the devnet
HEIMDALL_REPO="https://github.com/maticnetwork/heimdall.git" # repo of heimdall project
HEIMDALL_BRANCH=develop # heimdall branch defining the version to be used in the devnet
CONTRACTS_REPO="https://github.com/maticnetwork/contracts.git" # repo of contracts project
CONTRACTS_BRANCH=mardizzone/node-16 # contracts branch defining the version to be used in the devnet. To use node v16.x, please stick with it
GENESIS_CONTRACTS_REPO="https://github.com/maticnetwork/genesis-contracts.git" # repo of genesis-contracts project
GENESIS_CONTRACTS_BRANCH=master# Genesis contracts branch defining the version to be used in the devnet
MATIC_CLI_REPO="https://github.com/maticnetwork/matic-cli.git" # repo of matic-cli to run a specific version remotely using express-cli
MATIC_CLI_BRANCH=master # matic-cli branch used on the remote machines to start the environment
DEVNET_BOR_USERS=ubuntu,ubuntu,ubuntu,ubuntu # users' names of VMs for all the nodes (comma separated). Its length must be equal to "TF_VAR_BOR_VALIDATOR_COUNT + TF_VAR_BOR_SENTRY_COUNT + TF_VAR_BOR_ARCHIVE_COUNT"
DEVNET_ERIGON_USERS=ubuntu,ubuntu,ubuntu,ubuntu,ubuntu,ubuntu # users' names of VMs for all the nodes (comma separated). Its length must be equal to "TF_VAR_ERIGON_VALIDATOR_COUNT + TF_VAR_ERIGON_SENTRY_COUNT + TF_VAR_ERIGON_ARCHIVE_COUNT"
BOR_DOCKER_BUILD_CONTEXT="https://github.com/maticnetwork/bor.git#develop" # todo change to develop once https://polygon.atlassian.net/browse/POS-979 is solved (docker build context for bor. Used in docker setup (TF_VAR_DOCKERIZED=yes))
HEIMDALL_DOCKER_BUILD_CONTEXT="https://github.com/maticnetwork/heimdall.git#develop" # docker build context for heimdall. Used in docker setup (TF_VAR_DOCKERIZED=yes)
VERBOSE=true # if set to true will print logs also from remote machines
DD_API_KEY=DATADOG_API_KEY # Datadog API key

# Bor, Erigon and Heimdall Snapshot related variables (when running mainnet/testnet node)
BOR_SNAPSHOT_URL=https://snapshot-download.polygon.technology/bor-mumbai-fullnode-2023-03-18.tar.zst # URL for bor data snapshot (refer: https://snapshot.polygon.technology/)
HEIMDALL_SNAPSHOT_URL=https://snapshot-download.polygon.technology/heimdall-mumbai-fullnode-2023-03-18.tar.zst # URL for heimdall data snapshot (refer: https://snapshot.polygon.technology/)
ERIGON_SNAPSHOT_URL= # URL for Erigon data snapshot (refer: https://snapshot.polygon.technology/)

#Stress test variables (used to run stress tests against the remote nodes)
MNEMONIC="clock radar mass judge dismiss just intact mind resemble fringe diary casino" #random mnemonic
SPEED=200 # TPS = ~2 * SPEED (Default SPEED = 200; TPS = ~400)
MAX_ACCOUNTS=100000 # number of transactions to be sent
FUND=true # boolean flag to fund the accounts or not (use true for first time and false for subsequent runs)
STRESS_DEBUG_LOGS=false # boolean flag to enable debug logs for the stress tests

#EIP-1559 test variables
BURN_CONTRACT_ADDRESS=0x000000000000000000000000000000000000dead # Burn contract address
MAX_FEE=30000000009 # Max fee per gas
MAX_PRIORITY_FEE=30000000000 # Max priority fee per gas
COUNT=10 # Number of times to execute the test

#Shadow fork related variables
SHADOW_CHAIN_ID= #Chain Id of the shadow node (leave empty to get random one)

#RPC test related variables
RPC_URL= #External node URL to run the tests against (leave empty to run tests against a newly spun up node)
RPC_NETWORK= #The network for which tests will be run ('mumbai' or 'mainnet')
PRIVATE_KEY= #Private key of the account from which funds will be used for eth transaction calls
EXECUTION_COUNT_GETTERS= #The number of times the getter rpc calls are to be executed
EXECUTION_COUNT_SENDERS= #The number of times the sender rpc calls are to be executed

.env.example Outdated Show resolved Hide resolved
@anshalshukla anshalshukla merged commit 89be3cf into maticnetwork:master Aug 2, 2023
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants